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SUMMARY 

Our research activities consist of three overlapping areas 1) Information thMrymd^timia 
filtering are extended rigorously to visual information acqmsrtion and process.^ Th “ 
provide a comprehensive methodology for quantitatively assessing the end-tmend “ 

imaffe Catherine; coding, and processing. Information theory allows us to establ sh upper limits o 
the visual information that can be acquired within given constraints. °P t1 ^ 1 te ™ g ^ k 

establish upper limits on the performance that can be attained for specific tasks, even if these tasks 
relre aZtTve or interactive processing. 2) Focal-plane processing techniques and technology 

are developed to combine effectively image gathering with coding. The familiar 

vision processing akin to the retinal processing in human vision. Our approach includes the fami 

lateral inhibition, the new intensity-dependent spatial summation, and P^fj le se ^ sing / pr ° g 
networks. 3) A breadboard adaptive image-coding system is being assembled. This 
be used to develop and evaluate a number of advanced image-coding technologies and techniques 
£ well as toresearch the concept of adaptive image coding. The idea of adaptive image coding 
is to use knowledge-based supervision to autonomously select the appropriate data compressi n 
scheme based on (a) the properties of the acquired image data (and changes in these properties) 
b) the vTual information" required by the investigator and (c) the aval able ^ cap ^ Th 
system would have continual access to the investigators’ knowledge of what is significant. Ln der t 
investigators’ supervision, it could adaptively select image-coding and feature-class^ 
based on properties of the target such as spatial structure, texture, and spectral reflectan . 

1. INFORMATION THEORY AND OPTIMAL FILTERING 

The performance of (digital) image-gathering systems is constrained by the spatial-frequency 
response^ of optical apertures, the sampling passband of photon-detection mechanisms, and 
noise generated by photon detection and analog-to-digital conversion. Biophysical imitations 
have imposed simL" constraints on natural visiom Visual information is ^mevi ab y ost m both 
image gathering and low-level vision by aliasing, blurring, and noise. Therefore, it is no longe 
permissible to fssume sufficient sampling as Shannon and Wiener could do in their ( dasacad w orks, 
respectively, on communication theory and optimal filtering for tune-varying signals. N^ertheless 
the digital-processing algorithms (for image restoration, edge enhancement, etc.) found in the 
currently prevailing literature assume sufficient sampling, whereas image-gathering systems are 
orffinarily "designed to permit considerable insufficient sampling. This fundamental difference 
between ^assumption and" reality has caused unnecessary limitations in the P erfo = e oi ^ 
image gathering, coding, and processing. It also has led to unreliable conclusions about the 
correct ^design of image-gathering systems for visual information processing (as opposed to lm g 
reconstruction withou! processing 6 e.g., commercial TV) and about the actual performance of image- 
coding schemes for tasks which involve digital image processing. 

Our analyses so far have shown that the combined process of image gat ^” ng ‘ *“jj e °£L 
nrocessing (see fig. 1) can be treated as a commumcation channel if (and only if) the un g 
gathering 6 degradations are correctly accounted for. Correctly restored images gain significantly in 
Idehty (simifarity to target), resolution (minimum discernible detail), sharpness (contrast between 
large Leas) and Y clarity S ( absence of visible artifacts). These improvements in visual quality are 
obSnedTolely by the correct end-to-end optimization without increase in either data transmission 
m pTclsing These results haye encouraged us to extend our analyses to various image coding 
schemes and the associated image-restoration and feature-extraction algorithms. 
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Figure 1. Model of image gathering, digital processing, and image reconstruction. 


This work is summarized in references 1 through 6. 
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2. FOCAL-PLANE PROCESSING TECHNIQUES AND TECHNOLOGY 


Image gathering and coding are commonly treated as tasks separate from each other and from 
the digital processing used to restore and enhance images or extract features such as primal sketches 
or contour outlines. However, if we implement the edge enhancement required to draw primal 
sketches with focal-plane processing by properly combining optical response with lateral inhibition 
(as depicted by figs. 2 and 3), then a number of advantages can be gained. These advantages include 
improved resolution by a factor of four, improved robustness to noise, reduced data processing by 
two orders of magnitude, and reduced data transmission by nearly one order of magnitude. The 
implementation of a sensor-array focal-plane processor with lateral inhibition is supported by an 
SBIR contract (NASl-18287, Phase II). If this approach could be extended to include not only edge 
enhancement but also edge detection so that only significant primed sketches need to be transmitted, 
then it would be possible to reduce data transmission by two orders of magnitude without loss of 
the improved accuracy in edge location. 



Lens 



Figure 2. Model of optics and sensor array with focal-plane processing. 
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Optical design, Sensor-array with Combined response 

including lens apodization 3 * 3 lateral-inhibitory processing 


Figure 3. Laplacian-of-Gaussian response created by optical design and lateral-inhibitory processing. 


Our major emphasis is on the development of an image-coding 5 method that , we i refer ‘om local 
intensity adaptive image coding. This work is supported by two SBIR contracts (NAS1-18664 and 

NAS1-18850, both Phase II). 

Local intensity adaptive image coding consists of an innovative model of processing in the human 
retina referred to as Intensity Dependent Spread (IDS) and some additional logic to extr^t contour 
outlines and reflectance ratios at the boundary of two surfaces. Figure 4 is a schematic represe 
7Z IDS modd The line of detectors represents a slice through a two-dimensional array of 
detectors When an optical image or light distribution falls on the detector array, en eac e 
sends a signal into a network, where it spreads out. Each channel in ‘urn, sends ^ 
the sum of all the signals that arrive in its location m the summation network. The special property 
S^heTOS model hL to do with the way the signal from each detector spreads m ^ — on 
network As depicted in the lower half of figure 4, the magnitude of the signal at its center is 
proportional to the intensity of the light falling on the detector and the spread o f the signal 
inversely proportional to this intensity. The total volume under the spread remains constant Tha 
is all there is to the model. It has been demonstrated, in reference 8, that this simple space- van a 
model of image processing has many of the properties of human visual perception. 


iiumimmi 
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Detectors 

Summing network 
Output channels 



Detector output pattern: 
Constant volume 


Figure 4. Schematic representation of the IDS model. 


Figure 5 shows the response of the IDS processor to a spot, or point source, that is b ^te r th^ 
the background and to a step-type edge. Each detector spreads its signal as governed by the intensity 
of the light falling on the detector. For example, all of the spreads for the uniform back^und inj- 
ure 5(a) g axe the slme except for the one detector that is more brightly lighted. Its spread is higher and 


narrower. 
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(a) Impulse response (b) Edge response 


Figure 5. The IDS response to a point source and a step-type edge. 


Each output channel just adds up all of the contributions it receives. The result of this processing 
is shown as the output signal. As can be seen, the IDS response to a point source has a similar 
shape as the response of Marr and Hildreth’s familiar Laplacian of Gaussian (y 2 G) operator for 
enhancing edges (see fig. 2). In fact, the IDS processor exhibits center-surround antagonism and ail 
other manifestations of bandpass filtering that have made the y 2 G operator a favorite algorithm 
for low-level vision processing. However, the IDS response is nonnegative and spatially variant As 
we will show m the next three figures, the IDS processor accounts for several familiar perceptual 
p enomena of human vision that make it a highly robust low-level vision operator. 


First let us compare the IDS operation to conventional imaging. Figure 6 shows intensity 
profiles taken across conventional and IDS images of a step-type edge input for three illuminations. 
Conventional image-gathering yields a blurred representation that is visually representative of the 
target if the signal-to-noise ratio (SNR) is sufficiently high. As the illumination decreases, the 
representation gets buried in the noise. Image gathering with the IDS processor yields a target 
representation that consists of pulses. The one-crossing of each pulse locates the position of an 
edge in the target. The peak and trough values of the pulse are proportional to the ratio of the 
reflectances at the two sides of the edge, entirely independent of illumination. As the illumination 
decreases, the width of the pulse becomes broader (thereby trading resolution for sensitivity), but 
the accuracy of the one-crossing is unimpaired. For machine vision, this property means that edge 
detection for determining structure is highly robust in widely variant illumination. 


Next let us compare the IDS operation to edge detection with the linear y 2 G operator as well 
as to conventional imaging. Figure 7 shows intensity profiles taken across conventional images and 
outputs from the y G and IDS operators for two illuminations, high and low. Noise is disregarded 
or Simplicity. The peak and trough values of the y 2 G pulses are proportional to both illumination 
and reflectance. Therefore, it is not possible to characterize the reflectance properties of the target 
independent of illumination. However, the peak and trough values of the IDS pulses are proportional 
only to the reflectance changes. This striking property of the IDS processor mimics human visual 
perception (Weber s law) and makes it possible to extract the reflectance ratio at the boundary of 
two areas. J 
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DECREASING ILLUMINATION 



Figure 6. Comparison of IDS operation to conventional imaging. 



Figure 7. Comparison of IDS operation to conventional imaging and S7 2 G operation. 


Figure 8 illustrates a particularly important characteristic of the IDS filter, namely, the robust- 
ness of its reflectance representation to local variation in illumination (e.g., shadow). The recovered 
target 8(d) resembles the original one 8(a) and not the shadowy one 8(b), which is the one that 
was filtered. Traces of the shadow degradation can be seen in the modest loss of accuracy in the 
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actual transition as the illumination decreases. An important extension of IDS processing would be 
to extract color. Color then could be correctly detected independent of illumination. 



(a) Original target 
with constant 
illumination 



(b) Original target with 
spatially varying 
illumination 



(c) IDS response 
to target (b) 



(d) Reflectance 
recovery 


Figure 8. IDS response to target with spatially varying illumination. 

Odetics, Inc., is now under contract with LaRC (NASI- 18664, Phase II) to develop a hardware 
implementation of the IDS processor (see fig. 9). This processor will be capable of handling image 
data at real-time TV rates (30 frames per second). It will be implemented on several boards for 
the DATACUBE of Sun image- processing work stations. These boards are expected to become 
commercially available in Fall 1989. 



Figure 9. Implementation of IDS for real-time operation in Sun workstation. 

The full potential of the IDS processor for data compression as well as image enhancement and 
feature extraction is realized, of course, only when it is implemented as a focal-plane processor, or 
“retinal camera,” depicted in figure 10. The present design of the IDS processor for Sun workstations 
could be implemented in one 5 ’ by 5” board with 8 VLSI chips. A more advanced approach is the 
parallel asynchronous focal-plane image processor depicted in figure 11. This processor, which is 
being developed under a contract with LaRC (NASl-18850, Phase II), is representative of a new 
class of devices that permit full two-dimensional parallel readout and processing perpendicular to 
the focal plane. Advantages over conventional image-gathering and processing techniques include 
rapid parallel distributed processing, high dynamic range, and the elimination of conventional charge 
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S^rrSTS. visual pattern recognition. However, the development of this approach is s.,11 
in its initial experimental stage. 



Figure 10. IDS focal-plane processing camera. 


• 2-D parallel distributed data flow perpendicular to focal plane 
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• Low power, low noise integrate-and-fire devices for low level 
current-to-frequency conversion in every processing channel 
and at every network mode 


Input 



Figure 11. Parallel asynchronous focal-plane image processor. 
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This work is summarized in references 7 through 12. 


7. R. Alter- Gartenberg, C. L. Fales, F. O. Huck, and J. A. McCormick: “End-to-end 
performance of image gathering and processing for edge detection,” Computer Vision 
Graphics and Image Processing, submitted. 

^ T* Cornsweet and J. I. Yellott, Jr., “Intensity-dependent spatial summation,” 

J. Opt. Soc. Am. 2, pp. 1769-86 (1985). 

9. R. Alter-Gaxtenberg and F. O. Huck, “Feature extraction from intensity-dependent spatial 
summation. J. Opt. Soc. Am. A, submitted. 

10. R. Alter-Gartenberg, R. Narayanswamy, and K. S. Nolker, “From primal sketches to 

„ r e „ recov 1 ery of mtensit y reflectance representations,” Visual Information Processing 
Williamsburg, VA, 10-12 May, 1989. 


11. D. D. Coon and A. G. U. Perera, “Integrate-and-fire coding in Hodge-Huxley circuits 
employing silicon diodes,” Solid State Electronics 31, 851, 1988. 

12. D. D. Coon and A. G. U. Perera, “Integrate-and-fire dynamics and spike train information 
coding in neuron equivalent circuits employing silicon diodes,” Neural Networks 1, 1988. 


3. ADAPTIVE IMAGE CODING SYSTEM 


The problem of image gathering and data compression for the high-resolution, high-frame-rate 
video system is such that no single data-compression method, or even just a few methods, can 
satisfy most of the diverse requirements. For some experiments the data-compression requirements 
even change with time. Consequently, we have started to investigate the concept of adaptive image 

C jj 1 ? S ' ,p!,^ veStlsatl0n 1S su PP° rte d by one SBIR contract (Phase I). We are currentlv seeking 
additional SBIR contracts under the subtopic 07.01 entitled Focal Plane Image Processing. 

Figure 12 presents a block diagram of the basic adaptive image coding system. The idea of 
this system is to use knowledge-based supervision to autonomously select the appropriate data 
compression scheme based on (a) the properties of the acquired image data (and changes in these 
properties), (b) the visual information required by the investigator, and (c) the available channel 
capacity. The system would have continual access to the investigators’ knowledge of what is 
significant. Under the investigators’ supervision, it could adaptively select image-coding and feature- 
classification schemes based on properties of the scene such as spatial structure, texture, and spectral 



Figure 12. Schematic depiction of adaptive image coding system. 
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For example, consider figures 13 and 14. In one extreme, if the target of mte rest t is well 

smm ssrsss 

data compression to be continuously adjusted. 

could be autonomously controlled by the channel capacity that is available for transmitting 


desired images. 



Figure 13. Data compression for well illuminated target. The compression ratio varies from CR - 22:1 to 92:1 for 
8 the original color pictures shown here only in black and white. (Courtesy of Optivision, Inc.) 

In the other extreme, consider the scene shown in figure 14. An extreme range of light intensity 
exists stretching from deep shadow to highly reflective surfaces in direct sunlight. It is now important 
Aments of the astronaut in deep shadow, so an entirely different coding scheme is 
appropriate. The results shown in figure 14 were obtained with the locally adaptive coding .met 
based on the IDS model of retinal processing described in Section 2 above. Again, 

SSTto traiou. only the data paired for primal sketches a. very high data compress, cos. or 
more complete data that permits the recovery of reflectance representations. 
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Figure 14 Feature extraction under conditions of extreme variations of light 
to highly reflective surfaces in direct sunlight. 
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